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ABSTRACT 

The validity of assumptions that portfolios 
complement other assessment methods and yield more reliable and valid 
data than do traditional methods was studied. More specifically, 
focus was on examining whether: achievement level (characterized by 
scores, ranks, and group membership) varies signlt'icantly with 
differing definitions and usage of portfolios; and progress 
determined through portfolio assessment differs from progress 
ascertained using a variety of traditional assessment methods (such 
as paper and pencil tests, interviews, an*! cumulative grade point 
average) . Data on 30 college students in teacher edi'cation enrolled 
in three schools of the Alabama Consortium for Minority Teacher 
Education were collected during the freshman year and at the end of 
the sophomore year via standardized tests, questionnaires, and 
portfolios. Portfolios included students' exhibits of milestones 
reached, actlvitle.*, that they considered enriching to personal, 
social, and academic development. Data were analyzed using the 
Pearson correlation, multiple regression, and discriminant analysis 
techniques. Results show that performance measured using the 
portfolio technique Is related to performance determined using 
traditional measures such as ACT (American College Test), college 
grade point average, and essays. However, the nature of that 
relationship suggests using portfolios as a supplement, rather than 
as a substitute, to traditional measures. Two tables present study 
data, and a 28-item list of references is Included. (SLD) 
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ABSTRACT 



"WHAT TYPE OF EVIDENCE IS 
PROVIDED THROUGH THE PORTFOLIO 
ASSESSMENT METHOD?" 



Two assumptions underlie the use of the portfolio assessment 
technique. One '.s that portfolios complement other assessment 
methods. This is implied when portfolios are used in conjunction 
with other methods to document learning. Here, portfolios 
improve the reliability and validity of evidence. The second 
assumption is that portfolios yield more relevant and valid data 
than traditional methods. This is evidenced when portfolios are 
used as an alternative to traditional methods as is sometimes 
done when evaluating general education or teaching. which 
assumption is tenable has never been formally addressed. The 
current study investigated the validity of these assumptions. 
Specifically, it sought to find out whether portfolios yield more. 
encompassing evidence than, or tap abilities unrelated to those 
measured by, conventional methods. It will take more than this 
study to determine the correct assumption. The current study 
however, provides a starting point for dialogue i.i this area. 

The sample comprised 30 students in the Alabama Consortium 
for Minority Teacher Education. Data were collected by using 
standardized tests, questionnaires and portfolios. The data were 
analyzed using the pearson correlation, mul^tiple regression and 
discriminant analysis techniques. 

Results showed that performance measured using the 
portfolio technique is related to performance determined using 
traditional measures such as ACT, college GPA and essays. 
However^ the nature of that relationship suggests not to use 
portfolio, defined narrowly to exclude pencll-and-paper tests, as 
a substitute to traditional measures but rather as a supplement. 

The study provided some information on portfolio assessment 
that may help users determine when best to use this assessment 
technique. 
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The Portfolio assessment technique has become very popular 
In educational evaluation In recent times because of 
dissatisfaction with, for a variety of reasons, traditional 
assessment methods. Despite its growing popularity, portfolio 
assessment remains an enigma In many quarters. Nevertheless, 
It has been used for many evaluation purposes. Educators and 
researchers make either explicit or Implicit assumptions about 
the reliability and validity of portfolio assessment method when 
they use it. The purpose of this paper is to draw attention to 
these assumptions and to hopefully start a dialogue that will 
eventually help determine which assumption is more tenable than 
the other or when each assumption might be tenable. The next 
section will give a brief overview of the meaning of portfolios 
and and how they have been used in the literature. 

Portfolio, for many researchers (for example Collins, 1990), 
is a collection of documents assembled over a period of time by 
an examinee/candidate to provide evidence of his/her competence, 
knowledge, skills, abilities, and dispositions in relevant 
fields. Other users, however argue that portfolios are more than 
containers. Wolf and the TAP staff decided on a definition of 
portfolio that embraces its function as a depository of artifacts 
but, more Importantly, requires a written reflection by the 
developer on the significance or contributions of those artifacts 
to the attributes of Interest. Portfolio assessment is, thus, a 
method of assessment that is used to evaluate one's ability, 
knowledge base, skills, attitudes and dispositions through 
portfolio evidence. As with other measurement techniques, 
effective use of the portfolio assessment requires that each 
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portfolio be developed to serve a specified purpose. This 
purpose determines the contents of a portfolio. As Bird (1990) 
said, if purpose is not specified portfolio becomes a hodgepodge 
of documents. 

Portfolios have been used for various purposes. Their most 
popular uses seem to be in the assessment of general education 
outcomes (Black, 1990; Forrest, 1990; O'Brien, 1990; Hunter, 
1990); reading and/ or writing across the curriculum (Jongsma, 
1989; Elbow & Belanoff; (See Table 1 for more examples.)), 
credits for college-level skills and knowledge gained through 
work, volunteer activities, independent reading, military or 
corporate training, and other life experiences (Dagavarian, 1989; 
Thomson, 1988; Preston, 1981; Wolf, 1989; College of Boca Raton, 
Universities of Miami and Toledo) . 

The title of Forrest et al's paper portrays their assumption 
about portfolio assessment — Portfolio~assisted assessment of 
general education (1990) According to these writers, portfolio 
assessment is one of the most meaningful v«iys to utilize several 
assessment approaches in tracking students and assisting student 
learning. Here portfolio is seen as a conglomerate of various 
documents some of which may be results on pencil-and-paper tests 
or classroom observation. Defined this way, the underlying 
assumption is that judgments based on such a por folio is more 
reliable and valid because the evidence is more inclusive ani 
comprehensive . 

Portfolio asessment has also been used a great deal in 
measuring prior learning especially in adult education and 
alternative certification programs (Knapp & Gardiner, 1981; 
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Heerman, 1982; Woodrow, 1989; and Thomson, 1988). Most of these 
situations Involved assessing skills and competence through 
portfolio dor imentatlon In lieu of formal assessment, when the 
portfolio Is used this way, the underlying assumption Is that the 
decision made Is as reliable and valid as, or even more reliable 
and valid than, traditional tests which were considered 
inadequate to capture whatever is being measured. 

Similarly, portfolio assessment technique has been used to 
measure academic progress and/or "value-added" gains (O'Brien, 
1990; Nweke, 1990). Portfolios have also provided evidence for 
evaluating programs (Shaw, 1989; & Slevin, 1989); for advising 
and motivating students (Mattson-Sonoma, 1989; Mills-Court, 
1989), and for preparing a supplement to a resume (Nweke, 1990.) 

In addition to the above areas, portofollo assessment has 
made its debut into teacher assessement (Capie et al, 1979; Bird, 
1990; Shulman, 1989; Wolf, 1990). shulman and his staff at the 
Teacher Assessment Project (TAP) examined among other things the 
feasibility of using the portfolio assessment method to document 
and evaluate the complex art of teaching which they argue cannot 
and have not been effectively done by existing pencil-and-paper 
tests and observation methods. Shulman (1988) argues that a 
complete teacher assessment plan should Include evidence from 
four sources — pencil-and-paper tests, classroom observation, 
assessment center exercises, and documentation of performance in 
the form of portfolios. He contends that relying on any one of 
these sources is essentially Invalid. The TAP staff criticize 
such tests as the old NTE tests for ignoring the context and 
requiring one correct response and one best practice from every 
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teacher (Aburto and Nelson-Barber, 1987) . It is their conviction 
that the portfolio assessment method (and other TAP exercises) 
will provide teachers "the opportunity to describe and defend 
their personal teaching and evaluation strategies" (Aburto and 
Nelson-Barber, 1987). They emphasize, as did Jhulman, that 
portfolios can only be a supplement, not a replacement for, 
traditional measurement techniques. 

Shulman and the TAP staff make an important distinction 
among performance portfolios, on the one hand, and self-report, 
essays, test results, and letters of recommendation or 
recognition written by others, on the other hand. The contents of 
performance portfolios include what Vavrus and Collins (1988) 
call artifacts ( e.g teachers' handouts, tests, notes, samples of 
student ' work) , reproductions of nonportable and nonpermanent 
evidence such as pictures of bulletin boards, or blackboards, 
videotapes of classroom teaching, or an audiotape of planning 
sessions, and explanations and reflections attached to the 
entries showing how the documents relate to instruction. This 
type of portfoHo is performance oriented in contrast to some 
other types of portfolio that contain only self-reports, results 
of tests and examinations and essays on self-assigned or 
designated topics. 

Whether or not the portfolio can be used as a supplement or 
a substitute for traditional assessment techniques should depend 
on how the portfolio is defined and, consequently, the type of 
evidence included in it. The tenability of the implicit 
assumptions of the reliability of portfolios thus depends on how 
comprehensive or inclusive the contents of the portfolio are. A 
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review of the use of the portfolio in the literature shows that 
some educators include results of pencil and paper tests and 
examinations in addition to samples of writings over a period of 
time (See Table 1) . Defined and used this way, portfolios provide 
multiple sources of evidence for assessment and evaluation and 
thus can be argued to be conceptually more valid in terms of 
content or 



Insert Table 1 about here 



behaviors covered, the construct (s) measured and perhaps, 
relationship to specified criteria. Where portfolios contain 
essays or projects and exclude results from pencil and paper 
tests, portfolios may be tapping only a limited set of behaviors, 
skills and dispositions. This may be valid in the criterion- 
referenced interpretation sense if the set of behaviors or skills 
etc is representative of the domain of interest but, perhaps, not 
as a general measure of level of achievement. 

The first focus of this pa:.»er is to examine whether 
achievement level, characterized by scores, ranks or group 
membership, varies significantly with differing definitions and 
usage of the portfolio. The second focus is to investigate 
whether progress determined through portfolio assessment differs 
from progress ascertained using a variety of traditional 
assessment methods such as pencil and paper tests, interviews, 
cumulative grade point average, etc. 

Methodology 

Sample . The subjects for the study are 30 students, in the 
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first cohort of Alabama Consortium for Minority Teacher Education 
(ACMTE) , enrolled in teacher education programs in three of the 
seven institutions that make up the consortium. 

Procedure. The ACMTE espouses the value-added philosophy of 
measurement. Under this orientation, baseline measures were 
obtained on the students during the freshman year beginning with 
structured interviews, essays on assigned topics, Learning and 
Study Skills Inventory (LASSI), high school GPA, ACT or SAT, and 
numerous demographic data. Tho essay and interview were rated by 
at least two judges/ raters. The criteria for rating the Alabama 
English Language Proficiency test (AELPT) were uesd for grading 
the essays. The interviews were rated according to four criteria. 
These measures would serve as bases for determining how much 
'value' has been 'added' to the student, in other words, how much 
progress the students have made over any given period of time. 
Addit^ional multiple measures of students' achievement and 
progress are obtained in the sophomore year, some through a 
second administration of earlier-mentioned tests such as ACT, 
essays and interviews, others from first-time administration of 
other tests such as the AELPT, and yet other measures are 
obtained from portfolios, cumulative CPA, and acceptance or 
rejection decision into professional teacher education programs 
at the end of sophomore year. 

Some of the above measures are obtained from standardized 
tests and so the scores are directly comparable among the four 
schools. Grades from interviews, essays and portfolio, though 
based on the same scoring key, are less comparable because of 
variability in the graders/raters. We will bear this in mind 
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while examining any results from the study. Portfolio, for this 
study (not the the project) excludes traditional measures. 
Specifically, portfolios included students exhibits of milestones 
reached, activities participated in that they consider enriching 
to their social, personal and academic development. In addition, 
only a selected number of variables mentioned above will be 
included in the analyses for this paper. 
RESULTS AND DISCUSSION 

Data were analyzed using a variety of statistical tests. 
Table 2 shows the Intercorrelations among portfolio scores and 
other measures used in the study. Portfolio performance has its 
highest correlation (r= 0.504) with essay written at sophomore 
level despite very low relationship (r = 0.016) with essay 
written at freshman level. Portfolio performance shows a curious 
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and negative relationship with ACTl (r = - 0,115), ACT2 (r = 
-0.165) and the second interview (r=-0.206). it appears that 
portfolio taps something different from what the ACT's and 
interviews meas ire though it has quite a lot in common with the 
essay written after two years in college. The fact that the 
portfolio score is a measure of progress, and not mere level of 
achievement, may explain portfolio performance's higher 
relationship to the second essay, second interview and ACT2, than 
the first essay, first interview and ACTl (See Table 2). 

Also a multiple regression analysis shows that the best 
predictors of performance on the portfolio were current college 
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grade point average (GPA) , ACT2, second essay and second 
interview. These independent variables account for 41.7% of 
variability in portfolio performance. Similarly, a discriminant 
analysis was used to verify any variability in group membership. 
The results show that groupings based on discriminant functions 
formed by other measures such as performance on the essay written 
at the sophomore level, interview at the end of sophomore, :\CT 
and cumulative GPA were not very different from groupings based 
on portfolio performance. Specifically, 73% of the students were 
classified correctly using the discriminant function, while 27% 
were misclassif ied. 
CONCLUSION 

The results of this study, though tentative, suggest that 
the portfolio assessment may be quite a useful source of evidence 
in student assessment. Nevertheless, it also appears that 
caution needs to be exercised in using it. Specifically, the 
portfolio, which excludes traditional measures, may not be an 
adequate replacement for the latter as some authors have done. 
When defined narrowly (to exclude pencil-and paper tests) the 
portfolio should be used as a supplement, one of many sources of 
evidence, but not as a substitute, to traditional testing. 
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Table 1 - Summary of Users and Purposes of Portfolio Assessment* 



Purpose 


Users 


Assumptions 


General Education 


lo Forres i99u 

2. O'Brien, 1990 

3. Hunter, 1990 

4. Univ. of the 

City of New 
York 

5. Black, 1990 


supplement to 
traditional 
evaluation 
(Supplement) 

Alternative to 

portfolio 

(Substitute) 


Reading/ or Writing 
Proficiency 


Arzt, St. Joseph 
College 

Elbow and Belanof 
J ongsma , o ? 
Larson 

Lord, Plymouth 
State College 
Slevin, College of 
William and Mary 
Tebo-Messina, 
Withrop College 


Supplement 

Substitute 
•> 

• 

Substitute 
Supplement 

•> 

• 

Substitute 


Credit for non- 
formal education 


1. Dagavarian, 1989 

2. Daiker, Miami 

3. Preston, 1981 

4 . Ross 

5. Swift, Univ. of 
Toledo 

6. Thomson, 1988 

7. Wolf, 1989 


Substitute 
Substitute 

Substitute 
Substitute 
Mixed 

Substitute 
Supplement 


Prior knowledge - 
Adult education 


1. Heerman, 1982 

2. Knapp & Gardner, 
1981 

3. Thomson, 1988 

4. Woodrow, 1989 


Substitute 
Substitute 

Substitute 
Substitute 



Table 1 - Continued 



Teacher 
Assessment - 
Performance 
Portfolio 


1. Aburto & 
Nelson - Barber, 
1987 

2. Bird, 1990 

3. Capie et. al. , 

1979 

4. Shulman, 1988, 
1989 

5. Wolf, 1990 


Supplement 

supp 1 emen u 
Supplement 

Supplement 

Supplement 


Resume 


1. NweKe, 1990 

Alabama 

Consortium for 
Minority Teacher 
Education 
(ACMTE) 


supp 1 emen u 


Progress: Value- 
Added Education 


1. Nweke, 1990 
(ACMTE) 


Supplement 


Progress: Writing 


1. Rosenberg 

2. Larson, CUNY 


Supplement 
Substitute 


Advising & Self- 
Motivation 


1. Mattson-Sonoma 
State Univ. 

2. Mills-Courts, 
SUNY 


? 

Supplement 


Specific field 


1. Scott-Kenyon 
College 


Substitute- 
Reflection as a 
historian 


Program Evaluation 


1. Shaw - Univ. of 
Virginia 

2. Slevin - College 
of William & 
Mary 


Substitute 

• 









* This classifTcatTor^i^nany cases T^TaseS only on secondary 
sources of information and thus the details available may not 
have been sufficient to make an accurate classification. If 
this is the case for &xiy of the users listed in table 1, the 
author apologizes and will be glad to receive accurate 



information. A question mark (?) has been used to indicate 
total absence of information regarding how portfolio is used. 
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Table 2 Correlation Matrix 





HSGPA 


CGPA 


ACTl 


ACT2 


ESSl 


ESS 2 


INTl 


INT2 


CGPA 


0.439 
















ACTl 


0.454 


0.204 














ACT2 


0.370 


0.201 


0.860 












ESSl 


0.381 


0.411 


0.495 


0.480 










ESS2 


0.350 


0.227 


0.097 


0.110 


0.408 








INTl 


0.530 


0.414 


0.279 


0.227 


0.096 


0.373 






INT2 


0.349 


0.216 


0.401 


0.359 


0.329 


0.273 


0.498 




PTF 


0.213 


0.114 


-0.115 


-0.165 


0.016 


0.502 


0.072 


-0.206 



HSGPA - High school GPA 

CGPA - College GPA 

ACTl - ACT taken at Freshman level 

ACT2 - ACT taken at Sophomore level 

ESSl - Essay written at Freshman level 

ESS2 - Essay written at Sophomore level 

INTl - First interview during admission (Freshman) 

INT2 - Second interview into professional teacher education 

(Sophomore) 
PTF - Portfolio 



